The senses in an increasingly digital world

We interact with the world around us with all our senses—such as sight, hearing, smell, but also much more! The senses are fundamental to our experiences. The research area of multisensory experiences considers the different human senses and their interactions when designing human experiences. This area is growing in academic fields such as Human-Computer Interaction, marketing, and the arts, but also marks a growing trend beyond the academic world. Many practitioner initiatives are now placing the human senses at the centre of their design process such as chefs, marketers, and technology developers.

But what exactly are multisensory experiences?

We define multisensory experiences as “impressions formed by specific events, whose sensory elements have been carefully crafted by someone.” For instance, to create the impression of a sunflower, colours, textures, and specific smells are considered. The senses are situated at the centre of the formation of the impression, even in the absence of a real flower. That is, although you may not have a real flower, you may still use specific sensory elements to evoke the impression of the sunflower.

Though most of our experiences can be described as multisensory, multisensory experiences are different in that there is intentionality in them. While, say, walking in a forest or jungle involves several sensory elements (specific colours, aromas, textures), a multisensory experience is carefully designed by someone—a walk in the woods that has been designed by a landscape architect in order to evoke specific impressions, for example.

Multisensory experiences are increasingly transformed by advances in technology

Multisensory experiences are increasingly changed and enabled through technology. The sensory elements that are crafted in an event can be physical, digital, or a combination of both (mixed reality). Multisensory experiences move along the reality-virtuality continuum, where they can go from real through mixed reality (e.g. augmented reality) to fully virtual (e.g. virtual reality).

Let us consider the following example. A group of friends have a new project of growing sunflowers and they want you to experience them and they have three options to do so: (1) take you through their sunflower field without any technology, (2) take you through the field with the aid of augmented reality (AR) to obtain information about the sunflowers, or (3) take you through the field in virtual reality (VR). In the first, there may not be technology, in the second technology augments the experience, and in the third the experience is created through technology. Here, technology can influence the event or become the means for creating the event itself.

The future of multisensory experiences

Considering the ever-growing technological advances and increased understanding of the human senses and experiences, how may multisensory experiences be used in future to tackle some challenges that individuals and societies face, such as exploring new planets beyond Earth? Imagine a future where life has shifted from Earth to outer space, onto a new planet or solar system. What would life be like? How would we experience these new environments, through some or all our senses, or more?

Global efforts to prepare humanity for life beyond Earth can be observed and although it is not known yet what those future “outer space” experiences will be like, they will be different. There are efforts to make the interaction with the astronaut’s surrounding as fluent as possible exploiting all the human senses. For example, the concept of “earth memory bites” is designed to help astronauts have a shared eating experience of their favourite Earth foods together with their friends from Earth or people on the spaceship. This concept involves small bites that contain distinct flavours from different regions and cultures, or even flavours that one remembers from a specific event on Earth such as a wedding or birthday. By combining eating with VR and AR technology, it is possible to eat together with friends and family remotely. Music, atmospheric light, temperature, and humidity can also be used to recreate distinctive Earth settings. Emerging multisensory technologies makes it increasingly possible to imagine and create such experiences.

Possibilities and responsibilities when designing multisensory experiences

In recent times, there has been a growing digitization of human experiences, accelerated due to the COVID-19 pandemic. Purely offline and real experiences have shifted to mixed and virtual reality experiences where both the physical and digital worlds merge. However, many of the senses are still left “unsatisfied”, making it clear that there is still much to be done to create multisensory experiences in an increasingly digital world.

The excitement about multisensory experiences opens a plethora of opportunities but also challenges that need to be met with responsibility. Indeed, some of those challenges refer to how many businesses, individuals, and communities may not be ready for a digital transition and may potentially be left behind. In addition, issues that are already in the public’s eye, such as privacy, security, universal vs. exclusive access to technology, increased predictability, and controllability, will only become more and more salient.

Considering this and our definition of multisensory experiences, we have developed three laws of multisensory experiences, inspired by science fiction author Isaac Asimov and his three laws of robotics.

Our three laws focus on acknowledging and debating publicly different questions that are at the heart of the definition of multisensory experiences, namely, the why (the rationale/reason), what (the impression), when (the event), how (the sensory elements), who (the someone), and whom (the receiver), associated with a given multisensory experience.

The first law “Multisensory experiences should be used for good and must not harm others” particularly aims to guide the thinking process related to the question: what impressions and events do we want to design for and why? What is good or bad is not for one person to decide but must be subject to public debate. The second law “Receivers of a multisensory experience must be treated fairly” encourages reflection about who are we designing for, with the important reminder that different individuals need careful considerations, as a child needs to be treated differently from an adult. Finally, the third law “The someone and the sensory elements must be known” aims to address the specific question of who is crafting the multisensory experience and how that someone needs to ensure transparency of the use of sensory elements in the creation, especially when the someone cannot just be an individual, but also a company, maybe a government body, or an AI system.

Even if there are challenges, there are also great opportunities. As Henry David Thoreau noted, we also believe that “This world is but a canvas to our imaginations”. By considering and reinventing the multisensory worlds in which we live in, we can be the artists that paint the experiences that we imagine.

Featured image by Jessica Lewis via Pexels